Microprocessor verification is an important part and the bottleneck of the microprocessor design process. There is no practical method to test and verify all functional states of a complex microprocessor. A variety of techniques are used to generate test programs (i.e. tests) and to use these to verify the design's intended functionality. These include deterministic tests manually generated by engineers, tests generated by automatic test generators, such as IBM's Genesys and GenesysPro random test generators, standard test benches, or a combination of all techniques. In all cases, test programs are simulated against the design model to identify faulty functions.
One major issue that must be addressed is whether all functions of the processor have been tested and/or verified. If not, which ones have been missed. This information should be available in a timely manner so resources can be directed to verify functions that have been missed.
It is also very important to know promptly which functions have already been verified so that duplications are reduced and resources are targeted towards untested functions. Verification resources include verification engineers, test generators, simulation computers' capacity (i.e. simulation cycles) and debugging resources, just to name a few.
As the complexity of integrated circuit designs increase, so do the number and complexity of test programs to be generated and simulated. This, in turn, requires more resources to simulate and debug the functions that fail. The following questions need to be answered in order to decide whether the design has been verified or not:
1) Has everything detailed in the test plan been verified. If not, what are the remaining functions to be verified (functional coverage holes);
2) Have all new scenarios/states not planned for in the test plan been verified; and
3) Where should the attention/resources be focused to ensure the best coverage of verification.
To answer these questions, functional coverage tools and methodologies have been developed that monitor the simulation processes and analyze the test programs, thereby generating a vast amount of data on what has been tested so far. In addition, some coverage analysis tools/methodologies allow the verification procedures to define models of classes of scenarios to be verified and/or enumerate all design states to be verified. These, in turn, result in “coverage holes” reports, which can tell what scenarios/states have not yet been simulated/verified. Verification procedures can then focus on the remaining verification holes.
The simulation process generates a large amount of data, which needs to be reviewed and processed each day on a continuous basis (24/7). Automatic random test generators running on a large network of computers can generate hundreds of thousands of tests each day resulting in several gigabytes of test data that must be filtered and analyzed each day. In addition, coverage models and analysis filters applied to the above tests generate gigabytes of functional coverage data each day. In many cases, the simulation data that is generated and its corresponding coverage data cannot be fully analyzed as fast as they are generated, resulting in:
1) A failure to extract all of the important coverage data from simulation database, thereby having an incomplete view of what has been simulated;
2) A failure to properly interpret the accumulated data to provide correct feed back to verification process based on accurate and complete coverage data;
3) Shallow analysis of simulated tests because coverage models/filters are not extensive enough nor do they deal with the tiered and temporal functional features and design constraints;
4) Many interesting functional scenarios and patterns go undetected because they are not reflected in the coverage models and, hence, not detected;
5) No data is generated or considered on correlations between models, since models are developed independently and coverage analysis tools and methods work with the output of models and no cross-model analysis capability is usually provided;
6) No new discovery is made based on the actual simulation or coverage data because the generated tests or coverage data are not analyzed as a whole from a temporal point of view or overall design progress point of view but rather in isolation one test at a time and one model at a time; and
7) Not all design changes are reflected in coverage models, since models are defined based on the test plans and design specs rather than the actual design itself.
Consequently, the verification bottleneck is now spread to test and coverage analysis as well. Not only is the verification an incomplete task, but also the coverage analysis is incomplete and requires additional resources of its own.
Focused and targeted verification coverage analysis requires dynamic adjustment of the analysis process based on a deep knowledge of test features and coverage analysis data, taking into consideration the trends and relevant patterns of the actual verification progress, as well as any known and expected patterns missing in the collected data. Unfocused coverage analysis generates reams of data and many shallow coverage reports, which usually do not reflect the actual attributes of the verification process. Expertise for development of coverage analysis models and effective utilization of coverage tools are scarce. Designers and verification engineers who are the best candidates to write effective coverage models and analyze coverage data because of their knowledge of the design's internal workings and details are usually too busy with their own traditional tasks. Therefore, a new and dynamic coverage analysis process is required.
A way is needed to convert gigabytes of data collected each day to useable knowledge. In addition, hidden attributes, trends, patterns, associations and relationships in the collected verification and coverage analysis data that may shed light on the actual behavior and progress of the verification process is usually not considered. Coverage analysis tools generate a variety of reports, presenting predefined coverage information, which usually show accumulated statistics rather than shifts in the trends or new behavior. Intelligent and methodical analysis based on deep knowledge of the verification process is required to turn continuous flow of test and coverage analysis data to readily usable and relevant information. What is required is design intelligence (DI) and verification intelligence (VI) rather than more data and more reports.